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DENSITY AND LOCALIZATION OF RESONANCES 
FOR CONVEX CO-COMPACT HYPERBOLIC 

SURFACES 

FREDERIC NAUD 



p,i Abstract. Let X = F\H 2 be a convex co-compact hyperbolic 

surface. We show that the density of resonances of the Laplacian 
& ' Ax in strips {a < Rc(s) < 6} with |Im(s)| < T is less than 

Q(rpi+S-e(<r)j with £ ( a j > o as long as a > |. This improves 
the previous fractal Weyl upper bound of Zworski [28] and is in 
agreement with the conjecture of Q3] on the essential spectral gap. 



1. Introduction and results 



In this work, we will focus on the distribution of resonances of the 
Laplacian for a class of hyperbolic Riemann surface of infinite volume. 
The spectral theory of these objects can be viewed as a relevant pic- 
ture for more realistic physical models such as obstacle or potential 
scattering, but is also interesting in itself because of its connection 
with counting problems and number theory (see for example the recent 
work of Bourgain-Gamburd-Sarnak [I], or Bourgain-Kontorovich [5]). 

C^- \ Resonances replace the missing eigenvalues in non-compact situations 

and their precise distribution and localization in the complex plane is 
still a widely open subject. Let us be more specific. Let H 2 denote 
the usual hyperbolic plane with its standard metric with curvature —1, 
and let X = F\M 2 be a convex co-compact hyperbolic surface (see next 
§ for more details). Here T is a discrete group of isometries of HI 2 , con- 
vex co-compact (that is to say finitely generated, without non trivial 

^ ! parabolic elements or elliptic elements). In this paper we will assume 

that T is non- elementary which is equivalent to say that X is not a 
hyperbolic cylinder. The limit set A(r) is commonly defined as 

A(r) := Rinse 2 , 

where T.z denotes the orbit of z e H 2 under the action of V. This limit 
set actually does not depend on the choice of z and is a Cantor set in the 
above setting. We will denote by 8(F) the Hausdorff dimension of A(r). 
Let Ax denote the hyperbolic Laplacian on the surface X. The spectral 
theory on L 2 (X) has been described by Lax and Phillips [H] :[l/4, +oo) 
is the continuous spectrum, has no embedded eigenvalues. The rest of 
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the spectrum is made of a (possibly empty) finite set of eigenvalues, 
starting at 8(1 — 8). The fact that the bottom of the spectrum is related 
to the dimension 8 was first pointed out by Patterson [J_H] for convex 
co-compact groups. This result was later extented for geometrically 
finite groups by Sullivan [26] 125]. 

By the preceding description of the spectrum, the resolvent 

R x (s) = (A x - 8 (1 - s))" 1 : L 2 (X) -> L 2 (X), 

is therefore well defined and analytic on the half-plane {Re(s) > |} 
except at a possible finite set of poles corresponding to the finite point 
spectrum. Resonances are then defined as poles of the meromorphic 
continuation of 

R x (s) : C?(X) -* C°°(X) 

to the whole complex plane. The set of poles is denoted by 1Z X . This 
continuation is usually performed via the analytic Fredholm theorem 
after the construction of an adequate parametrix. The first result of this 
kind in the more general setting of asymptotically hyperbolic manifolds 
is due to Mazzeo and Melrose [TB]. A more precise parametrix for 
surfaces was constructed by Guillope and Zworski [T^J [TT] . Note that 
by the above construction and choice of spectral parameter s = a + it, 
the resonance set (including possible eigenvalues) TZ X is included in 
the half-plane {Re(s)<5}. If 8 > |, then except for a finite set of 
eigenvalues, 1Z X C {Re(s) < |}. 

One of the basic problems of the theory is to localize resonances with 
the largest real part, which are resonances who play a key role in var- 
ious asymptotic problems, including hyperbolic lattice point counting 
and wave asymptotics. Another central and related question is the ex- 
istence of a fractal Weyl law when counting resonances in strips. More 
precisely, in the papers by Zworski and Guillope-Lin-Zworski, jTOj [28] 
they prove the following. For all a < 5 set 

N(a,T) := #{z eK x : a < Re(z) < 5 and < Im(z) < T}, 

then for all a, one can find C a such that for all T > 0, one has 

N(a,T)<C a (l + T 1+s ). 

The first upper bound of this type involving a "fractal" dimension is 
due to Sjostrand [23] for potential scattering, see also [51] . In [TO] , 
the authors give some numerical evidence that the above estimate may 
be optimal provided a is small enough, but no rigorous results so far 
have confirmed this conjecture, and the best existing lower bound is a 
sublinear omega estimate, see [9]. On the other hand, it is conjectured 



DENSITY AND LOCALIZATION OF RESONANCES 3 

in [T3] that for all e > 0, there are only finitely many resonances in the 
half-plane 

Re(s) > - + e 

hence suggesting that one has to take a < 5/2 to observe the optimal 
rate of growth of N(o~, T) as T — > +oo. In this paper, we show the fol- 
lowing which is in agreement with the previous conjecture and provides 
a more precise picture in term of the distribution of resonances. 

Theorem 1.1. Let V be a non- elementary convex co-compact group 
as above. Then for all a > |, one can find r(cr) > such that as 

r-> +oo, 

N(a,T) = O (T 1+r(,T) ) , 

where r(|) = 5, r(cr) < 5 for all | < a. Moreover, one can find 
cr > | such that the map a i-> r(cr) is real- analytic, convex and strictly 
decreasing on [|,cr ]. In particular, its derivative at a = | satisfies 
r'(|)<0. 

The main achievement of that paper is that r(a) < 6 for all a > | and 
says that the 0(T 1+<5 ) upper bound is non-optimal whenever a > |. 
From a physics point of view, this result is in agreement with the idea 
that the density of resonances has to "peak" in a strip close to "half 
of the classical escape rate" which is exactly | in our setting. See 
the paper [15] for some numerical study that support this conjecture, 
especially figure 2. We also refer the reader to [18] for a comprehensive 
survey on questions related to fractal Weyl laws and spectral gaps for 
various open (chaotic) quantum systems. An "explicit" expression for 
t(o~) is provided in the last section (see formula &§), involving the 
topological pressure of the Bowen-Series map. The above result is of 
course relevant for a close enough to the critical value | since we know 
from a previous work of the author [T7] that there always exists a 
spectral gap i.e. one can find e > such that 

U x n {Re(s) >5-e} = {5}. 

Unfortunately, this e is hardly explicit. Of course if 5 > |, then this 
spectral gap is already given by Lax-Phillips theory, but the above 
theorem is still meaningful since | < |. 

This result can also be compared with the spectral deviations ob- 
tained by N. Anantharaman [T] for the resonances of the damped wave 
equation on negativey curved manifolds, except that in our case no 
Weyl law is known rigorously. The techniques we rely on to prove 
Theorem 11.11 are therefore very different although they also involve 
some ergodic theory and thermodynamical formalism. We would like 
to point out that Theorem 11.11 has a natural interpretation in terms of 
Selberg zeta function. Let us denote by V the set of primitive closed 
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geodesies on X = T\M 2 and given C G V we denote by 1(C) its length. 
Selberg zeta function Z r (s) is usually defined through the infinite prod- 
uct (Re(s) is taken large enough) 

Z r (s):= J] (l- e -(*+ fc X( c )). 

(C,fc)e-PxN 

It is known since the work of Patterson-Perry [2D] that this function 
extends analytically to C and the non-trivial (non topological) zeros 
of Zr(s) are the resonances. Therefore Theorem ll.il can be read as a 
statement on zeros of Selberg's zeta function, and this is actually how 
we prove it. Let us make a few comments on the organization of the 
paper. In the next section we recall the transfer operator approach for 
Selberg's zeta function. In section §3 we prove the necessary a priori 
bounds to control the growth of a (modified ) Selberg zeta function in 
strips, generalizing slightly the method of [TO]. In §4 we use a classical 
lemma of Littlewood to relate the counting function for resonances to 
a "mean square estimate". The idea of §6, which is the core of that 
paper, is to exhibit some cancellations in the quadratic sums to beat 
the pointwise bound of |10j . To achieve this goal some lower bounds 
on the derivatives of "off-diagonal phases" are required and proved in 
§5. The technique we use in §6 bears some similarity with the work 
of Dolgopyat [H [T7] and can be viewed as an "averaged" Dolgopyat 
estimate. We would like to mention that the ideas presented here 
should extend without major difficulties to higher dimensional Schottky 
groups, actually only §5 needs to be significantly modified. 

2. Transfer operator and Selberg's zeta function 

We use the notations of §1. Let H 2 denote the Poincare upper half- 
plane H 2 = {x + iy G C : y > 0} endowed with its standard metric of 
constant —1 curvature 

2 = dx 2 + dy 2 

y 

The group of (positive) isometries of H 2 is naturally isomorphic to 
PSL2(M) through the action of 2 x 2 matrices viewed as Mobius trans- 
forms 

az + b j l 
z i—)- :, ad — be = 1. 

cz + d 
A Fuchsian Schottky group is a discrete subgroup of PSL 2 (M) built 
as follows. Let Pi, ... , V p , V p+1 , . . . , V 2p be 2p Euclidean open discs 
in C orthogonal to the line R ~ dM 2 . We assume that for all % ^ j, 
T>i fl Vj = 0. Let 71, . . . , 7 P G PSL 2 (M) be p isometries such that for all 
i = 1, . . . ,p, we have 

where C := C U {oo} stands for the Riemann sphere. 
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The discrete group T generated by 71 , . . . , j p and their inverses is called 
a classical Schottky group. If p > 1 then T is said to be non elementary. 
It is always a free, geometrically finite, discrete group and if in addition 
we require that for all i 7^ j,T> i C\'Dj = 0, then T is a convex co-compact 
group i.e. the quotient Riemann surface 

x = r\e 2 

is an infinite volume geometrically finite hyperbolic surface with no 
cusps. The converse is true, up to an isometry, all convex co-compact 
hyperbolic surfaces can be uniformized by a group as above, see [TJ. 
Let T C PSL 2 (M) be a Fuchsian Schottky group as defined earlier: 

where 7i(£>i) = C \ V p+i . We also set for i — 1, . . . ,p, 7 p+i := 7" 1 . For 
all j = 1, ... , 2p, let H 2 (T>j) denote the Bergman space of holomorphic 
functions defined by 

H 2 {Vj) := I f : Vj -»■ C : / holomorphic and / \f\ 2 dm < +00 I , 

here m stands for the usual Lebesgue measure. Each function space 
H 2 {Vj) is a Hilbert space when endowed with the obvious norm. We 

set 

H 2 :=($H\V 3 ). 
i=i 
The Ruelle transfer operator is a bounded linear operator Jzf s : H 2 — > 
H 2 defined by (z G V u s e C ) 

with the notation / = (/1, . . . , / 2p ), and j + p is understood mod 2p. 
We have to say a few words about the complex powers here: we have 
l'j(J)i) cC\ M~ for i 7^ j so 7j(z) s is understood as 

where L is a complex logarithm on C \ Mr which coincides on M + \ {0} 
with the usual logarithm. For example, one can take 



Uz) 

'1 s 

This operator 5£ s acts as a compact, trace class operator on H 2 and 

we refer the reader to [31 [10] for a proof. The Fredholm determinant 

associated to this family is the Selberg zeta function defined above: for 

all s G C, we have 

Z r (s) = det(I-JSf 5 ). 
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This remarkable formula that dates back to the work of Pollicott 
in the co-compact case, is the starting point of our analysis. Unfortu- 
nately, the space H 2 as defined above is not enough for our analysis and 
we will need to use as in [TU] a family of Hilbert spaces H 2 (h) depend- 
ing on a small scale parameter < h < ho. We recall the construction 
taken from [10], see also [3] chapter 15. Let < h and set 

A(h):=A + (-h,+h), 

then for all h small enough, A(h) is a bounded subset of M whose 
connected components have length at most Ch where C > is inde- 
pendent of h, see [3] Lemma 15.12. Let Ie(h) denote these connected 
components, with £ = 1, . . . , N(h). The existence of a finite Patterson- 
Sullivan measure /i supported by A(T) plus Sullivan Shadow Lemma 
(see [3], chapter 14) show that 

A~ l h s N{h) <5>(itf0) = A*(A(r)), 

e 

for some uniform A > 0, hence the number N(h) of connected com- 
ponents is O (h~ s ). Given 1 < £ < N(h), let T>e(h) be the unique 
euclidean open disc in C orthogonal to M such that 

D,(/i)ni = /f(/i). 

Now set 

N(h) 

H*(h):=®H*(T> £ (h)). 

e=i 

We will see in the next section that for n > n (with n independent 
of h) that the operators J2? s n act as compact trace class operators on 
H 2 (h). Moreover, the Fredholm determinant 



r(n) 



det(/ - £> s n ) 



is a multiple of the Selberg zeta function and we will count resonances 
using this determinant instead of the original zeta function. The goal 
of the next section is to provide a proof of the following fact, which is 
a slight modification of the argument of Guillope-Lin-Zworski in |10j . 

Proposition 2.1. For all o"o < 5, there exists C > such that for all 
<t < Re(s) < 5 and n > n , we have for |Im(s)| large, 

log|4 n) (s)| <C\lm(s)\ 5 e nP ^°\ 
where P(o~) is the topological pressure at a. 
We refer the reader to the next § for a definition of topological pressure. 
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3. Basic pointwise estimates 

The goal of this section is to prove the previous Proposition. We 
first introduce some notations. We recall that 7^ . . . , j p are generators 
of the Schottky group T. Considering a finite sequence a with 

a = (ai,..., a n ) G {l,...,2p} n , 

we set 

7a :=7ai °---°7a„- 

We then denote by W n the set of admissible sequences of length n by 

W n := {a G {1, . . . , 2p} n : V i — 1, . . . , n — 1, a i+ i 7^ a« + p mod 2p}. 

We point out that if a G #^, then 7 a is a reduced word in the free 
group T. For all j = 1, . . . , 2p, we define Wj by 

Wj:={aeW n : a n ^j}. 

If a G Wj, then 7 Q maps "Dj into D 0l+p . Given the above notations 
and / G i^ 2 , we have for all z G T>j and n G N, 



^:u)(z) = E (-/«w)v(7aW). 



We will need throughout the paper some distortion estimates for 
these maps 7 Q . More precisely we have for all j = 1, . . . , 2p, 

• (Uniform hyperbolicity). One can find C > and < < 9 < 1 
such that for all n,j and a G y^f, 

C- 1 ^ < sup K| <C0 n . 

• (Bounded distortion) . There exists M\ > such that for all n,j 
and all a G Wj , 



sup 

£>,■ 



7" 

/a 



7;, 



< Mi. 



The second estimate, called bounded distortion, will be used constantly 
throughout the paper. In particular it implies that for all z±,Z2 G T>j, 
for all a G W^, we have 

e -\zi-Z2\Mi < halgjJI ^ e |«i-«a|Afi 
" |V a (*2)| " 

These estimates are rather standard facts in the classical ergodic theory 
of uniformly expanding Markov maps. For a proof of the first estimate, 
we refer the reader to [3] for example. Bounded distortion follows 
from hyperbolicity and elementary computations. Another critical tool 
in our analysis is the Topological pressure and Bowen's formula. Let 
Ij := T>j fl M. The Bowen-Series map T : U^Jj -^RU {00} is defined 



ev (»)(«) 
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by T(x) = 7i(x) if x G Ij. The non- wandering set of this map is exactly 
the limit set A(r) of the group: 

+00 

A(r) = f|^"(uE 1 /,). 

n=l 

The limit set is T-invariant and given a continuous map ip : A(r) — > R, 
the topological pressure P((p) can be defined through the variational 
formula: 

P{<p) = sup I h^(T) + / tpd/J, 

M V J A 

where the supremum is taken over all T-invariant probability measures 
on A, and h^iT) stands for the measure-theoretic entropy. A cele- 
brated result of Bowen [B] says that if one considers the map o 1— \ 
P(—cr log |T'|) then this map is convex, strictly decreasing and van- 
ishes exactly at o = 5(T), the Hausdorff dimension of the limit set. 
An alternative way to compute the topological pressure is to look at 
weighted sums on periodic orbits i.e. we have 

1/ra 

(1) e p ^= lim f V 

\T n x=x 

with the notation ip^ n >{x) = <p(x) + (p(Tx) + . . . + ip(T n ~ 1 x). We will 
seldom use the two previous formulas in our analysis but will rather 
use the following upper bound. For simplicity, we will use the notation 
P(cr) in place of P(— crlog \T'\). 

Lemma 3.1. For all a"o < M in M., one can find Cq > such that for 
all n > 1 and M > a > o~o, we have 

Proof. For all a > o~q and x G Ij, we have by bounded distortion 
property 

£ sup Kr < c m J2 (Y»r = c m j^ (i)(x). 

The Ruelle-Perron-Frobenius theorem [2] applied to j£? CTo : C 1 (J) — > 
C 1 (7) (/ = Ujlj) says that this operator is quasi-compact, has spectral 
radius e p ^ and e P( - <T0 - ) is the only eigenvalue on the circle 

{\ z \ = e p(CTo) }. 

Moreover, e p ^ ^ is a simple eigenvalue whose spectral projection is 
given by 

f^V(f):=ipoJfdv , 
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where /xq is a probability measure and tpo is a positive C l density on I. 
Using Holomorphic functional calculus, we can therefore decompose 

where J\f has spectral radius at most 9oe p ^ ^ for some 9q < 1. It is 
now clear that we have 

J2 sup WS < C sup^oe nP(CTo) + CJve nP{ -°°\ 

for some constants C , C\ and 9 < 9 < 1 and the proof is done. □ 

We can now go back to the action of 

if s n : H 2 (h) -)• H 2 (h). 

Set £j(h) := {1 < £ < N(h) : V e (h) C £>,-}. Given a G ^' and 
£ G Sj(h), we have provided n is large enough, 

la{Ve,{h)) C V e (h) 

for some £' G {1, . . . , N(h)}. Indeed, since the limit set A is T-invariant, 
7 Q maps T>j(h) into a disc of size at most Ch9 n which contains an 
element of A. Taking n large enough (uniformly on h), this disc has to 
be inside UiV^h). Therefore =Sf s n leaves H 2 {h) invariant as long as n 
is large. We will actually need something more quantitative which is 
proved in jTOj [3] : 

Lemma 3.2. There exists n , k > such that for all n > n , a G Wj[, 
we have for all I G $j{h), 7 a (^(/i)) C T>£i{h) with 

dist(-f a (V e (h)),dV e/ (h)) > nh. 

To estimate the pointwise growth of Zf' (s) = det(J — =Sf s n ) we will use 
some singular values estimates on the space H 2 (h) with h = |Im(s)| _1 . 
Notice that this zeta function does not depend on h. Indeed, provided 
Re(s) > 5, we have the formula 

(3) det(J - JS?) = exp (- f^ \^A^f) J , 

and the trace can be computed (using a fixed point analysis, see [3]): 

(T p )'(x)- S 



Tr(j^) = E T 



TPx=x vv I \ Jl 

the above sums over periodic orbits of the Bowen-Series are clearly 
independent of the choice of function space, hence of h which will be 
adjusted to optimize our estimates. First we need to recall some facts 
on singular values of compact operators and our basic reference is the 
book of Simon [22]. If % is a complex Hilbert space and T : % — > % 
is a compact operator, the singular values Hk(T) are the eigenvalues 



10 FREDERIC NAUD 



(ordered decreasingly) of the self-adjoint positive operator \/T*T . If 
we have 



X>co 



< +00, 
*;=i 

then T is said to be trace class. If T is trace class, then the eigenvalue 
sequence 

|Ai(T)|>|A 2 (T)|>...>|A*(T)| 
is summable and the trace Tr(T) is defined by 



Tr(T) = ^A fc (T), 
fc=i 

while the determinant det(J + T) is given by 

00 
det(J + r):=JJ(l + A fc (T)). 

fc=i 

Weyl's inequalities show that we have 

00 00 

n( i +i^( r )i)^ri( i +^( T ))- 
fc=i fc=i 

In this paper we will use the following remark: 

Lemma 3.3. Let T : Ti — > H be a trace class operator and (e^ £ N is a 
Hilbert basis ofH, then we have 

00 

log|det(J + T)|<E||T(e,)||. 

Proof. By Weyl's inequality, we write 

00 00 

log|det(/ + T)| <^>g(l + /^(T)) < Y,^(T) = Tr(Vf^T). 

k=0 fc=0 

But the classical Lidskii theorem says that 

00 00 

TriVf^T) = J2{VT*Te t ,e e ) < J^ \\VT*fe e \\, 

by Schwartz inequality, but we have || -\/T*Te^ || = ||Te^||. D 

We can now give a proof of Proposition 12.11 First we need an explicit 
basis of H 2 (h). Setting for £ e {1, . . . , N(h)}, 

D e (h):=D(c ei r e ) 
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with q61 and h < r% < Ch, we denote by e e k the function in H 2 (h) 
defined for z £ T>j(h) by 



if JV 



edz 



k+1 1 / 2-C 



1 J 



J \ 'J 



A- 



if J 



It is straightworfard to check that {^)kes,i<e<N(h) is a Hilbert basis of 
H 2 (h). We now use Lemma 13. 31 and write 

oo N(h) 
fc=0 1=1 

Using Schwartz inequality, we get {N(h) = 0(h~ s )) 

s oo /N(h) > 1/2 

(4) iogi4 n) (^)i < ch-2j2[ E \\^ n (4)\\W) 

k=0 \^=1 

On the other hand we have 

iV(ft) 2p JV(/i) 

1=1 j=i t>e£j(h)J' D i'( h ') i=i 

where m is the usual Lebesgue measure on C. Given £' £ ^(h), we can 
write 

(5) / J2\^U)\ 2 dm< J2 / 1^)11 W^>", 

where 

i=i 
We now need to prove the following remark. 

Lemma 3.4. Setting Qj{h) := U^ e ^.(M^(/i), we /iawe /or a// a,/3 £ 



" n i 



sup |F«| < Ch~ 2 p k , 

flj(h) 



with C and < p < 1 uniform 

C l JV cj, vvc uavc cituci ^ Q/3 



Proof. First remark that given z £ Vlj(h), we have either i^ i(z) = 



or 



^» = l e 2 , °7«WIKo 7 ^) 



for some £o £ {1, - - - , N(h)}. Then combine Lemma 13.21 with the ex- 
plicit formula for e fc ° to obtain the result. □ 
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Going back to estimate fl^D, and using the fact (use bounded distortion) 
that we set |Im(s)| = h~ x 

^p m s \<c\w a C;\ 

zev e ,(h) 
we have reached 

N(h) I 2p 

E W^ n H)\\m ih] < Ch~ 5 p k E E SU P Kl* 

< C/i-V fe e 2nP(<T) . 
The proof is done by inserting the above estimate in formula (J2J) and 
summing over /c. D 

4. Applying Littlewood's Lemma 



We now show how to reduce the proof of Theorem 11.11 to a mean 
square estimate on transfer operators. To this end we apply a result 
of Littlewood taken from the classics. More precisely, we prove the 
following. Define M(a, T) by 

M{a,T) := #{sen x : a < Re(s) < 5 and T/2 < Im(s) < T}. 

Proposition 4.1. Let a < a < 5, then for all T large and n(T) = 
[v log T] with v > small enough, we have 

rT 

M{a, T)<C X log |4 n(T)) (ao + it)\dt + C 2 T. 

JT/2 

Proof. We start by recalling a version of Littlewood's Lemma which 
suits our needs. Let a < 1 and T > 0. Let / be a function which is 
holomorphic on a neighborhood of the rectangle 

R T>ao := [o- ,l} +i[T/2,T], 

and assume that / does not vanish on the segment l+i[T/2, T]. Denote 
by Zt,ct q the set of zeros of / on -Rt> • Then we have the formula 

2vr E (Re(*) - <r ) = / hg\f(a + it)\dt- f log |/(1 + it)\dt 
+ J Arg(f(a + iT))da- f Kig{f{a + iT/2))da. 

J ao J (Jq 

The function Arg(/) is the imaginary part of a determination of a 
complex logarithm log/ which is defined by taking upper limits of 
a holomorphic logarithm on a suitable simply connected domain (see 
Titchmarsh [27], section 9.9, for more details). The resulting function 
Arg(/) is well defined on Rr,a \ Z and is discontinuous on a finite 
union of segments. 
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Fix o"o < er < 5 < 1. Then since IZx is a subset of the set of zeros of 
Zp (s), applying the above formula to Zp (s) we get 



27r(<7 - cr )M(o-,T) < / log |Z r n(T)) (a + it)\dt 

JT/2 

-of [ log |4 n(T)) (l + it)\dt) +o( sup |Arg(Z r n(T)) (a + iT))\ 

\JT/2 J \<T <a<l 



IT/2 

An(T)) 



+0 sup \ArgiZp 1 "(a + iT/2))\ 

\cr <cr<l 

Remark that combining flTJ and the trace formula (^ shows (-P(l) < 0) 
that Zp (s) is uniformly bounded on {Re(s) = 1}, hence 

/ \og\Z^ n{T) \l+tt)\dt = 0(T). 

JT/2 

To control Arg(Z r (s)), we will use another classical result of Titch- 
marsh essentially based on Jensen's formula, see [27], 9.4. 

Lemma 4.2. Fzx 2 > o"o > and let f be a holomorphic function on 
the half-plane {Re(s) > 0}. Suppose that |Re(/(2 + it))\ > m > /or 
all t 6 K and assume that f(s) = f(s) for all s. Suppose in addition 
that |/(cr + it)\ < Ao-j, then ifT is not the imaginary part of a zero of 
f(s), for alia > a , 

|Arg(/(<r + iT))\ < C ao (\ogA^ T+2 - logm) + 3tt/2. 

To be able to apply the above Lemma, we need to prove the following. 

Lemma 4.3. There exists m > such that for all n large enough, we 
have for all t, 

|Re(Z r n) (2 + 2t))| >m. 

Proof. First remark that for Re(s) > 6, 

( °° 1 \ /°° 1 

Re(det(/-^)) = exp - ^ -Re(Tr(^ fc )) cos PT -Im(Tr(^; 

V fc=i / \fc=i 

and the trace formula combined with the pressure formula ([T|) shows 
that for all e > and p large, 

|Tr(j^f)| < (7 e P0P(Ms))+e). 
We therefore obtain 

|Re(Z r n) (2 + it))\ > exp (-Ce n ^ 2 ^) \ cos (Ce n ^ 2 ^) |. 

Since P(2) < by Bowen's formula, we have a uniform bound from 
below as long as n is taken large. □ 
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To finish the proof of Proposition 14.11 we use the pointwise estimate of 
Proposition 12.11 which combined with Titchmarsh Lemma gives 

sup |Arg(4 n(T) V + *T))\ = O (T 5+uPM ) = 0(T), 

CTQ<(T<1 



as long as we choose v < p , , . The proof is done, provided that there 



1-6 

r. ± ne piuui is uuiie, pi 

are no zeros on {Im(s) = T} and {Im(s) = T/2}. If Z r (s) vanishes 
on these lines, we simply replace T by some T<T<T + lso that 
Zp (s) does not vanish on {Im(s) = T} and T/2 by some T' with 
T/2-1 < T' < T/2 so that Z r n(T)) (s) does not vanish on {Im(s) = T'}. 
We then write 

M(a,T) < #{s G ft x : Re(s) > a and T' < Im(s) < T} 

/•T 

<C log |Z r n(T) Vo + zt)|dt + 0(T) 

JT' 

rT 

= C log 1 4 n(T)) (o-o + «) I dt + O (T) , 

JT/2 

by the pointwise estimate of proposition 12.11 □ 

We now state the main estimate which is the core argument of the 
paper. Let (fo G Cq°(R) be a smooth compactly supported function 
such that Supp(<^ ) C [— 1, +1], tpo > on (—1, +1) and J tp (x)dx = 1. 
We define a probability measure \xt on R by the formula 

f 1 /" + °° ft — 2T\ 

J f d i* : = t J w(-r- )/(*)*■ 

Proposition 4.4. There exist u>0,0<p<l such that for all a > | 
one can /md e(er) > suc/i £/ia£ 

JVC) , 

£=1 ^ M 

where we have taken n(T) := [i/logT], /i = T _1 . 

The proof of this proposition is postponed to §6 and occupies the rest 
of the paper. Let us show how the combination of Proposition 14.41 and 
Proposition 14.11 implies Theorem 11.11 By Proposition 14.11 we have for 
2 < o-q < a, 

MK V) <C [' log 1 4" (T)) (<r + it)\dt + O(T), 

4 



and we write 



f 2 log\Z^ T)) {a + it)\dt 
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< ( inf <p Q ) T f log |4 n(T) Vo + it)W T {t). 

VJ-3/4,1/2] J J R 

Using Lemma 13. 3[ we obtain 

\og\Z^ T) \a + i t)\dMt)<J2H / \\^Shei)\\( h )dMt), 

k£N £=1 J R 

which by Schwartz inequality is less than 

Ch ~* E E / \\^S\4)\\hdMt) < cn*-*»w 

fceN yi=i ^ R J 

We have therefore obtained 

-rl+max^ S—-^e{ao),0 > 



M(a,T) = 0\T 
To get an estimate on the counting function N(cr,T), we just write 

No 



f T\ 

N(a,T)<J2MU-\+0(l] 



fc=0 
T 



where N is such that ^j < 1- Since we have for T large, 
with C independent on T, the proof is done. 

5. A KEY LOWER BOUND 

Given a,/3 e Wj, we set for all x E Ij = Vj n R, 

$ a> p(x) := log W a (x)\ - log |7J,(aO|. 

The goal of this section is to prove the following fact which will be a 
key result in the next section on mean square estimates. 

Proposition 5.1. There exists r/o > and 1 > 9 > such that for all 
n > 1 and all j — 1, ... , 2p, we have for all a ^ (3 E Wj , 

m \^(x)\ > m r . 

The proof of this proposition will follow from the next Lemma which 
is of geometric nature. 

Lemma 5.2. Let V be a Schottky group as above, then there exists a 
constant Cp > such that for all 

i^( a c b d )er\{id}, \c\>c r . 
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Proof. Let r 7^ Id be an element of T, and assume that 

a b 



c d 

with c = 0. By looking at 7 -1 one can assume that \a\ > 1. In addition, 
since T has no non-trivial parabolic element, we have actually \a\ > 1. 
The action of 7 on the Riemann sphere is therefore of the type 

7(2;) = a 2 z + ab. 

Now pick an element of A(T) which is different from the fixed point of 
7. Its orbit under the action of (7™) n eN goes to infinity as n — > +00 
which contradicts the fact that A is 7-invariant and compact. Therefore 
c^O. Remark that by definition of T, we have 

7 - 1 (oo) = -d/c e uf =1 T>j. 

Consequently, one can find a constant Q\ such that for all 7 7^ Id, 
\d/c\ < Q\. Notice also that we have (i = y/—l) 

1 
Im(7(z')) 



c 2 + d 2 

Since the limit set A(r) is compact, the orbit T.i has to be bounded 
in C for the usual euclidean distance. Hence there exists Q2 such that 

for all 7 G T, 

1 







c 2 


+ d^^ 


As a consequence we , 


get 










1 < 


c 2 (Q2 + Ql 


and the proof is done. 


□ 







We can now complete the proof of Proposition 15.11 Writing 
Ja{%) = f , with a a d a - b a c a = 1, 



we have 



\cpd a - c a d£ 



i*w = 2 , 'T; N ;;^i = 2\cpd a -c a dp\(iM) i/2 w^) i/2 - 

We can now remark that since T is a free group, a 7^ (3 implies 7 a °7^ x 7^ 
Id, and we have the formula 

7 07" 1 - ( aa ba )( d ? - b e ) = ( 

'p \ c a d a J \ -cp ap J \ c a dp - d a c p * 

We can therefore apply the above Lemma and the proof ends by using 
bounded distortion and the lower bound for the derivatives. 
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6. Mean square estimates 

The goal of this final section is to prove Proposition 14.41 We recall 
that in the sequel we take 

h = T~\ n(T) = [ulogT], < v « 1. 

We first start by writing 

N(h) 2p N{h) 

E / II^V*)fo<Mt) = E / / E \^P(4)\ 2 d»Tdm. 

1=1 ^R j=l J^j{h) JR e=1 

For all z G Vj and a, (3 G Wj we use the notation 



$ a ,^):=L( 7 ;(2))-L(7^)), 

where L is the adequate complex logarithm. This notation coincides 
on the real axis with the one introduced in the previous section. Re- 
mark that since $ Qj/ 3 is real on the real axis and because of bounded 
distortion, we have on any disc V^h), 

sup | e iT *^ (2) | =0(1), 

z£T> e (h) 

uniformly on T. For all z G Qj(h), we have by a change of variable 

N(h) 

Ei^V,)wr^ = 

e=i 
E (7;(^)) <T (%M) ,7 G'S(^)^(-^^(^))e 2jT# -' 3W , 

where ^o is the usual Fourier transform of (fio defined by 

/+oo 
if (x)e- ix ^dx, 
-oo 

and G a p is the following sum (see also Lemma E3]): 

G %( z ) '■= E e k ° 7a(*M o jp(z). 

The goal now is to gain some decay as T goes to +oo by using the key 
observation of Lemma 15.11 We split the above sum into two contribu- 
tions, the "diagonal" one plus the "off- diagonal" one: 

where we have set 
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y o f fdl a g := Yl (i(^)) ff (#)) ff <^)^(-^ a ^(^))e 2!T$ ^ (2) . 

We first deal with the diagonal contribution, which by bounded distor- 
tion and the pressure estimate (together with Lemma 13. 4p gives 

and therefore 

/ \y diag \dm < Ch~ s e n ^ p ^p k . 

JClj(h) 

We clearly have a gain as long as P (2cr) < 0, which by Bowen's formula 
[6] amounts to say that a > |. To deal with the off-diagonal sum, we 
will of course use the estimate for £ £ C and all q > 0, 



1^(01 <c q - 



e IMOI 



;i + l£lK 

which implies that for a^/3 and all q, we have 

\^ o (-m a , (z))\ = O q ((1 + \T$ a ^z)\)-«) . 

The trouble comes from <& a> p(z) which may be vanishing on some part 
of Vtjih). We prove the following Lemma. 

Lemma 6.1. Fix an e > 0. There exists v small enough (recall that 
n(T) = [vlogT]) such that for all a, (3 £ W£ with a ^ (3, one can split 
the set <gj{h) = gl U g'j so that: 

• We have#g( = 0(hT2). 

• For all i £ S" and z £ T> t (h), 

\^ a A^\>ch^ 5 2 +e . 

Proof. By Proposition 15.11 for all a ^ (3, the function x H- $ Q)( g(x) 
is strictly monotonic on the interval Ij, and its derivative is uniformly 
bounded from below by Ci(9) n . Two cases can occur: 

• Either <& a p(x) vanishes at some point x £ Ij and we set J a ^ = 
[xq — h v ,x + h v ], where < 77 < 1 will be adjusted later on. 
Then for all x £ Ij\ J a ,p we have 

|$«,/j(aOI> W(0) n . 

• The map x H- $ ajJ g(x) does not vanish on Ij := (dj,bj). Ac- 
cording to the sign of $ Qi/ g, we set J a> p = [aj,dj + h v ] or 
Ja,p = [bj — h v , bj\. In both cases we have for x £ Ij \ J a> p, 

\®aAx)\>Cih n (d) n - 
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Now define S'j' by 

%' = {ee£j(h) : V e (h)nJ a ^ = ®}. 

Given £ e <£", we have by bounded distortion for all z 6 T>i(h), 

I $«,/?(«) I > C x hP(9) n - C 2 h > C 3 W + \ 
provided t] + e < 1 and v\ log 0| < e. It remains to count 

4}:=4(h)\4». 

Except for possibly two of them, £ G &! implies RnP|(/i) C J aw s, hence 
by volume comparison 

{^' j -2)Ch<\J a ,p\ = N>. 

Therefore j^<§\ = C^/i 77-1 ). The proof ends by choosing r\ — 1 — ~. D 



Going back to S^ ffdiag , we have using Lemma 13.41 

dm(z) 



JtijQi) ^ri Jnj(h) l 1 + 1 1 ^a^yzjlr 

We then write 

dm(z) f dm(z) f dm(z) 



, n . (h) (1 + \T^ p (z)\Y J n , (h) (1 + \T^p{z)\Y J„, {h) (1 + \T^ p (z)\Y 
with the notations 

Q'^h) := |J ^(/i) and Q^/i) := |J P,(/i). 

We recall to the reader that 



ee&j i<^«" 



#£'. = 0(h~2) and #$'! = 0(/T 5 ). 
Applying the above Lemma and taking e > small enough so that 



2 e>0 ' 



5 

2 

we get with T = h" 1 and q large 

O { h 2 ^ )+0{ h 2 - s+ «2- e) 



dm(z) / „ i / „ ?, ,5 



Qj{h) 



:i + \T$ a Az)\y 



I h 2 ~l ) 



Therefore, 

/ \yoff d ia g \dm<Cp k e 2nP ^h-l. 

Jttj(h) 

Adding all of our estimates, we have reached 

E I II^V*)fe<M*) < c P k (r»» p ™ + T5+^w j 
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Now taking 

v < min 



6 5 



8|logflfl6P(f) 7 P(f) 



yields for all a > |, 

N(h) 

£ / ll^$V*)ll? h )<Mt) < ^P fc (T^(-) +T 35 / 4 ) . 

The proof of Proposition 14.41 is done. We can now say a few more words 
on the function r(cr) of the main Theorem ll.il using the above choice 
of v then for all a > a® > |, we can take 

r(<7) = max|<J + |P(2<r ),— 

so for example taking a = Mcr + |) gives 

(6) r((T ) = max | ( 5 + ^p( (T + |) 7 ^ 

We have not attempted to optimize the choice of constants at all. For- 
mula ([6]) shows in addition that r(a) is strictly decreasing and convex 
in a right neighborhood of | since the topological pressure has this 
property. Moreover, the pressure functional a *— > P{c) is real analytic 
and its derivative at a = 5 is given by 



P'(5) = - f log \T'\dfi s <0, 



where I = U^/j, T is the Bowen- Series map and fis is the equilibrium 
state at 5. In particular, this gives a formula for the derivative 



r'it 



~2 / lo § \ T 'W& 
which is of course non- vanishing. 
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